This is possibly just a small already over-repeated discussion, however, it is important to mention it to both state my stance regarding the matter, and act as a disclaimer and excuse for possibly bad grammar, vocabulary, and etc. Why I do NOT use any AI tools in this blog, not even LanguageTool or Grammarly, only a spell checker! And why am I writing this instead of an actual blog post? To be fair, grammar checkers are good, and my lack of use is mostly because of privacy concerns (I know local versions exist) and difficulty of setup. They also change your use of words, that I do not support for a personal blog; see here (404media) for more details.
The situation is at least serious enough so that the authors guild has announced a certificate for human authored books 1.
Let's answer the second question first. I'm in a taxi, and I've been thinking about how I would do graduate math after I get into a MSc program. Would I use these AI tools or not, and if I would, how much?
In the end, the main job is on our human brain. We do get rusty if we don't use it, and we get exhausted if we do. A natural occurrence. However, Using LLMs will make us more prone to these two factors. Of course, by giving up the freedom of thought to some degree, and allowing a tool to accept a part of our burden, we live easier, at the expense of our thinking ability. However, this reduced capacity, will result in earlier exhaustion as well. However, completely forgoing the tool would also affect the speed of your research. This is a double edged sword, and I am self-aware that unless this getting rusty proves to have any serious consequences, you can turn a blind eye to it, and I also know that my stance is because of something in the form of a meta-reason. I will explain.
Other than mental and intellectual effects of these tools, the quality of a human produced and reviewed content is still superior to AI generated content. Don't take me wrong, what it produces "is" good. However, almost any moment you want to actually create something new and out of ordinary, it doesn't perform well.
Finally, of course, there are some ethical aspects to this topic as well. This is a part of the reason I mentioned earlier. There are other products, such as wood table, leather tools, etc, that people prefer handmade items. Mathematics is the same in my opinion, and so is art, and so is a normal blog post. Even a simple spelling mistake that the writer makes, is a sign of he being himself. His choice of words, insight, and method he chooses to write, express, and solve. That has a value in itself 2.
To build on top of the last paragraph, even if there is a hypocrisy in western countries about pushing this technology forward, some AI companies themselves reject AI generated resumes, and I have seen them not welcome AI generated content in a few occasions. To be fair, this is a systematic issue, and as much as I am avoiding LLMs, this injustice is on the part of the society. Still, no matter the reason, it proves a point in longevity of human efforts, instead of pure technological advances.